Appendix A - MASTER

This Appendix describes the format of the MASTER file.

Each Record in the Master file is 53 bytes in Length. The first 53 bytes in the file contains some information regarding index numbers, and all remaining 53 byte records contain security information. The Master file record can contain no more than 255 security records.



The following shows the Byte Configuration for reading and writing to the MASTER File.

This shows the configuration for the first 53 byte record.

Field Name

Format

Start(Position Byte#)

Length(# of Bytes)

Function

# of Records

UB

1

2

Stores the # of records stored in the MASTER file

Next Record #

UB

3

2

Stores the last # used for a F#.dat file

Blank Space


5

48




The following shows the 53 Byte configuration for the security records which would consist of all remaining 53Byte records found in this file.

Field Name

Format

Start(Position Byte#)

Length(# of Bytes)

Function

Record #

UB

1

1

Stores the index # for the F#.dat file

File Type


2

2

This is always written as 25856(Int). Not sure what function this provides

Record Length


4

1


Record Count


5

1

# of 4 byte fields in the F#.dat file. Dependant on whether its Intraday, has Open Price, Open Interest and High and low.

Unknown


6

2


Issue Name


8

16

Description provide for the security

Unknown


24

1


CT_V2_8_FLAG


25

1

If this is 89(Int) then V28 is true, if this is 42(Int) then Autorun is True. (I have no clue what this is used for, but it doesn seem to affect anything)

First Date

MBF

26

4

This is the date defined when the security is created. Is used to determine how far back to capture security data.

Last Date

MBF

30

4

This is the date of the last quote on record.

Time Frame

A

34

1

This is the timeframe for the security defined when the security is created. 'I' for Intraday, 'D' for Daily, 'W' for Weekly, Etc.

Intraday Timeframe


35

2

This is the Intraday Time base defined when the security is created. 1Min,5min, 10min

Symbol

A

37

14

Ascii symbol defined when the security is created.

Unknown


51

1


Flag


52

1

Always set to 0.

Unknown


53

1




See MasterFileRecord.vb for further details on how each security record is read, and the functions used to convert the bytes to useful data. See Functions.vb for how a MasterByteArray is constructed. The MasterFileRecord.vb class uses the MasterFileDescriptor.vb to provide the byte locations for each record.

Appendix B - EMASTER

This Appendix describes the format of the EMASTER file.

Each Record in the EMaster file is 192 bytes in Length. The first 192 bytes in the file contains some information regarding index numbers, and all remaining 192 byte records contain security information. The EMaster file contains some duplicate data of the the MASTER, and like the Master file contains no more than 255 security records



The following shows the Byte Configuration for reading and writing to the EMASTER File.

This shows the configuration for the first 192 byte record.

Field Name

Format

Start(Position Byte#)

Length(# of Bytes)

Function

# of Records

UB

1

2

Stores the # of records stored in the EMASTER file

Next Record #

UB

3

2

Stores the last # used for a F#.dat file

Blank Space


5

187




The following shows the 192 Byte configuration for the security records which would consist of all remaining 192Byte records found in this file.


Field Name

Format

Start(Position Byte#)

Length(# of Bytes)

Function

ID Code


1

1


Record #


2

1

Stores the index # for the F#.dat file

Unknown


3

2


Record Count


5

1

# of 4 byte fields in the F#.dat file. Dependant on whether its Intraday, has Open Price, Open Interest and High and low.

Fields


6

1

Tells which fields are active, MSB- Time,OI,OP,HighorLow,HighorLow,Unknown,Uknown – LSB

Unknown


7

8


AutoRun


10

1

Unsure what this is used for

Unknown


11

1


Symbol

Ascii

12

13

Ascii symbol defined when the security is created.

Unknown


26

6


Issue Name

Ascii

33

16

Security description defined when the security is created.

Unknown


50

11


Time Frame

Ascii

61

1

This is the timeframe for the security defined when the security is created. 'I' for Intraday, 'D' for Daily, 'W' for Weekly, Etc.

Unknown


62

3


First Date

CVS - 4 byte single precision real

65

4

This is the date defined when the security is created. Is used to determine how far back to capture security data. YYMMDD

Uknown


70

3


Last Date

CVS - 4 byte single precision real

73

4

The is the date of the Last quote on Record. YYMMDD

Unknown


78

3


Start Time

CVS - 4 byte single precision real

81

4

Defines the start time for an Intraday Security

End Time

CVS - 4 byte single precision real

85

4

Defines the stop time for an Intraday Security.

Unknown


89

38


First Date Long

CVL - 4 byte long integer

127

4

This is the date of the Last Quote in Long Format YYYYMMDD

Unknown


131

1


Dividend Date

CVL - 4 byte long integer

132

4

Not sure how this is used.

Dividend Date

CVS - 4 byte single precision real

136

4

Not sure how this is used.

Unknown


141

52




See EMasterFileRecord.vb for further details on how each security record is read, and the functions used to convert the bytes to useful data. See Functions.vb for how a EMasterByteArray is constructed. The EMasterFileRecord.vb class uses the EMasterFileDescriptor.vb to provide the byte locations for each record.

Appendix C - XMASTER

This Appendix describes the format of the XMASTER file.

Each Record in the XMaster file is 150 bytes in Length. The first 150 bytes in the file contains some information regarding index numbers, and all remaining 150 byte records contain security information. The XMaster file is used after the MASTER and EMASTER have been populated with 255 security records, the Limit of the MASTER and EMASTER files. The XMaster can house Index Number 256 to 65535, and its respective information.



The following shows the Byte Configuration for reading and writing to the MASTER File.

This shows the configuration for the first 150 byte record.

Field Name

Format

Start(Position Byte#)

Length(# of Bytes)

Function

Unknown


1

1

Always seems to be 0x5D

Unknown


2

1

Always seems to be 0xFE

Unknown


3

2

Always seems to be Ascii “XM”

Unknown


5

6


# of Records


11

2

Number of records stored in the XMASTER

Unknown


13

2


# of Records


15

2

Number of records stored in the XMASTER(seems to be the same)

Unknown


17

2


Next Record #


19

2

Stores the last # used for a F#.dat file

Unknown


21

129




The following shows the 150 Byte configuration for the security records which would consist of all remaining 150Byte records found in this file.


Field Name

Format

Start(Position Byte#)

Length(# of Bytes)

Function

Uknown


1

1


Symbol

Ascii

2

14

Ascii symbol defined when the security is created.

Issue Name

Ascii

17

23

Security description defined when the security is created.

Unknown


41

1


Unknown


42

1

Always seems to be 0xFF. Unsure of function.

Unknown


43

5


Unknown


47

1

Always seems to be 0x7F. Unsure of function

Unknown


48

13


Time Frame

Ascii

62

1

This is the timeframe for the security defined when the security is created. 'I' for Intraday, 'D' for Daily, 'W' for Weekly, Etc.

Unknown


63

1


Record #


66

2

Stores the index # for the F#.dat file

Unknown


68

2


Fields


71

1

Tells which fields are active, MSB- Time,OI,OP,HighorLow,HighorLow,Unknown,Uknown – LSB

Unknown


72

7


Start Date

CVL - 4 byte long integer

81

4


Short Start Date


85

3


Unknown


87

16


First Date Long

CVL - 4 byte long integer

105

4

This is the date of the Last Quote in Long Format YYYYMMDD

Last Date

CVL - 4 byte long integer

109

4

The is the date of the Last quote on Record. Long Format YYYYMMDD

Unknown


113

3


Last Date

CVL - 4 byte long integer

117

4

The is the date of the Last quote on Record. Long Format YYYYMMDD (seems to repeat)

Unknown


121

29




See XMasterFileRecord.vb for further details on how each security record is read, and the functions used to convert the bytes to useful data. See Functions.vb for how a XMasterByteArray is constructed. The XMasterFileRecord.vb class uses the XMasterFileDescriptor.vb to provide the byte locations for each record.

Appendix D – F*.DAT (or F*.MWD)

This Appendix describes the format of the .DAT or .MWD file.

The .dat file contains a Dummy record at the beginning of the file. In the Dummy record the Number of quotes records stored .dat file are recorded/stored at the 3rd and 4th byte of that .dat file.

The byte configuration is a little flexible in that it changes depending on the # of Fields defined by the MASTER/EMASTER or XMASTER files. The “Fields” byte in the EMASTER or XMASTER determine how many 4 byte fields will make up a record.

In the case of Tick data a record could have as few as 4 by 4 byte fields making up a record which would consist of Date, Time, Close and Volume for a total of 16 bytes per record.

In the case of an Intraday Commodity, you could have as many as 8 by 4 byte fields making up a record which would consist of Date, Time, Open, High, Low, Close, Volume, Open Interest for a total of 32 bytes per record.



Each 4 byte record goes in the following sequential order : Date, Time, Open, High, Low, Close, Volume, Open Interest.

If any field is not present it is removed from the order in order to make up that particular record.

See the if statements in the LoadQuotes function in the Metafile.vb for the possible combinations.